Mission Success Starts With Safety 



Demonstrating the Safety and Reliability of a New System 
or Spacecraft: Incorporating Analyses and Reviews of the 
Design and Processing in Determining the Number of 

Tests to be Conducted 


William E. Vesely 
Alfredo E. Colon 

Office of Safety and Mission Assurance 
NASA Headquarters 
Washington, DC 20546 


10/2010 (1) 


Mission Success Starts With Safety 

Presentation Outline 

* Demonstrating Design Safety/Reliability 

* Failure Rate Bathtub Curve 

* Reliability-Growth-Based Testing Requirements 

* Treatment of Uncertainties 

* Benefits and Costs of Testing Strategies 

* Summary 

* Supplemental Slides 

* Annotated References 
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emonstrating Design Safety and Reliability 


• Design safety and reliability is the probability that a new 
system has no failure-causing faults 

• Design tests focus on detecting existing failure-causing 
faults 

• Design tests can be partial tests or complete system 
tests and can consist of test flights 

• In addition to design-related failures, random failures 
can occur 

• The random failure contribution is generally associated 
with the steady-state operation of the mature system 
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The Failure-Rate Bathtub Curve 
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Reliability-Growth Principles Provide 
Required Numbers of Failure-Free Tests 

• Design safety and reliability is demonstrated by 
conducting sufficient tests without failure 

• Based on reliability-growth principles, the required 
number of tests depends on three major factors:* 

- Initial System Assurance Level 

- Fault-Detection Effectiveness 

- Corrective Action Effectiveness 

• Failures are handled by including corrective action 
effectiveness in the test requirements 

• Binomial testing requirements are not applicable since 
failure correction and test feedback are not considered 



*See the references for further background 
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Factors Determining Initial Assurance Level 
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^ Factors Determining Fault Detection Effectiveness 
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Required Reliability-Growth-Based Failure-Free Tests* * 


• The three tables in the supplemental slides give required failure- 
free tests to demonstrate a given system reliability 

• The first table shows the value of having a high initial assurance 
with much fewer demonstration tests needed 

• The second table shows that inapplicable binomial testing 
requirements are generally much higher than reliability-growth- 
based testing requirements 

• The third table shows the effect of increased fault detection 
effectiveness in decreasing required numbers of tests 

• To include uncertainties, lower bounds on the initial assurance 
level and fault detection effectiveness are used 


*See the supplemental slides for the formulas and the references for further background. 
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jrtainty Evaluations for Large Uncertainties on the 
il Assurance Level and Fault Detection Effectiveness 


Initial Zero-Failure Assurance Uncertainty Distribution 


Test Effectiveness Uncertainty Distribution 


Zero-Failure Assurance Distribution After 1 Test 





N = 500000 Bandwidth = 0.01 507 


N = 500000 Bandwidth = 0.007528 


N = 500000 Bandwidth = 0.0149 


Zero-Failure Assurance Distribution After 2 Tests 


Zero-Failure Assurance Distribution After 3 Tests 


Zero-Failure Assurance Distribution After 4 Tests 





N = 500000 Bandwidth = 0.01442 


N = 500000 Bandwidth = 0.01 374 


N = 500000 Bandwidth = 0.01297 


Zero-Failure Assurance Distribution After 5 Tests 


Zero-Failure Assurance Distribution After 10 Tests 


Zero-Failure Assurance Distribution After 15 Tests 





N = 500000 Bandwidth = 0.0118 


N = 500000 Bandwidth = 0.004589 


N = 500000 Bandwidth = 0.001 578 
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Evaluating the Benefits and Costs of Subsystem and 
System Test Strategies 



System 

Reliability with 
Subsystem Tests 

System Failure 
Probability with 
Subsystem 
Tests 


System 

Reliability with 
Subsystem and 
System Tests 

System Failure 
Probability with 
Subsystem and 
System Tests 


Total 

Subsystem 
Test Cost 

Total System 
Test Cost 

Total 

Subsystem 
Plus System 
Test Cost 

Before Tests 

49.00% 

51.00% 


90.09% 

9.91% 


120 

300 

420 

After Tests 

90.09% 

9.91% 


98.64% 

1.36% 


























Subsystem 1 


Subsystem 2 


Subsystem 3 


Subsystem 4 


System 











PreTest 

Reliability 

70.00% 


70.00% 


100.00% 


100.00% 


90.09% 

Number of 
Tests 

3 


3 


1 


1 


3 

Test 

Effectiveness 

0.5 


0.5 


0.5 


0.5 


0.5 

Post Test 
Reliability 

94.92% 


94.92% 


100.00% 


100.00% 


98.64% 

Post lest 

Failure 

Probability 

5.08% 


5.08% 


0.00% 


0.00% 


1.36% 

Cost per Test 

20 


20 






100 

Total Test Cost 

60 


60 


0 


0 


300 
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Determining Reliability-Growth-Based 
Testing Requirements in Practice 


• The initial assurance level and fault detection 
effectiveness are assessed 

• The determining factors are assessed and are combined 

• Grading criteria are defined for each factor 

• Historical values are incorporated 

• Uncertainties are treated 

• Robust test requirements are determined 

• Required tests are based on applicable reliability factors 
and not on inapplicable binomial lot sampling tables 
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Initial Failure History from 1960 to 2000 for Space 
Launch Vehicles for the First Five Launches* 


Launch Number 

1 

2 

3 

4 

5 

Attempts 

41 

40 

38 

36 

34 

Failures 

13 

10 

6 

6 

7 

Mean Failure Rate 

0.32 

0.25 

0.16 

0.17 

0.21 

Standard Deviation 

0.471 

0.439 

0.370 

0.378 

0.410 

Bayesian Mean 

0.33 

0.26 

0.18 

0.18 

0.22 

95% Bayesian Interval 

(0.20, 0.47) 

(0.14,0.40) 

(0.08, 0.31) 

(0.08, 0.32) 

(0.10,0.37) 


*Seth D. Guikema and M. Elisabeth Pate-Cornell, “Probability of Infancy Problems 
for Space Launch Vehicles”, Reliability Engineering and System Safety 87, March 
2005, pp. 303-314 
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Summary 


* Design Safety/Reliability is Associated with the Probability 
of No Failure-Causing Faults Existing in a Design 

* Confidence in the Non-Existence of Failure-Causing Faults 
is Increased by Performing Tests with No Failure 

* Reliability-Growth Testing Requirements Are Based on 
Initial Assurance and Fault Detection Probability 

* Using Binomial Tables Generally Gives Too Many Required 
Tests Compared to Reliability-Growth Requirements 

* Reliability-Growth Testing Requirements are Based on 
Reliability Principles and Factors and Should Be Used 
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Supplemental Slides 
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Reliability-Growth-Based Failure-Free Tests to Demonstrate a 
Given Design Reliability Versus Initial Assurance Level 
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Initial 
Assurance 
Failure- 
Free Tests 

1 

5% 

10% 

15% 

20% 

25% 

30% 

35% 

40% 

45% 

50% 

55% 

60% 

65% 

70% 

75% 

80% 

85% 

90% 

95% 

Probability That the System is Failure-Free After Conducting a Given Number of Failure-Free Tests 

6.557% 

12.903% 

19.048% 

25.000% 

30.769% 

36.364% 

41.791% 

47.059% 

52.174% 

57.143% 

61.972% 

66.667% 

71.233% 

75.676% 

80.000% 

84.211% 

88.312% 

92.308% 

96.203% 

2 

8.556% 

16.495% 

23.881% 

30.769% 

37.209% 

43.243% 

48.908% 

54.237% 

59.259% 

64.000% 

68.482% 

72.727% 

76.753% 

80.576% 

84.211% 

87.671% 

90.970% 

94.118% 

97.125% 

3 

11.092% 

20.847% 

29.493% 

37.209% 

44.138% 

50.394% 

56.070% 

61.244% 

65.979% 

70.330% 

74.340% 

78.049% 

81.489% 

84.688% 

87.671% 

90.459% 

93.071% 

95.522% 

97.828% 

4 

14.262% 

25.990% 

35.804% 

44.138% 

51.303% 

57.528% 

62.988% 

67.815% 

72.113% 

75.964% 

79.436% 

82.581% 

85.443% 

88.059% 

90.459% 

92.670% 

94.712% 

96.604% 

98.362% 

5 

18.153% 

31.890% 

42.649% 

51.303% 

58.414% 

64.362% 

69.410% 

73.749% 

77.517% 

80.821% 

83.741% 

86.341% 

88.670% 

90.769% 

92.670% 

94.400% 

95.981% 

97.431% 

98.766% 

6 

22.823% 

38.435% 

49.787% 

58.414% 

65.192% 

70.657% 

75.158% 

78.929% 

82.134% 

84.891% 

87.289% 

89.393% 

91.255% 

92.913% 

94.400% 

95.740% 

96.955% 

98.061% 

99.072% 

7 

28.279% 

45.427% 

56.934% 

65.192% 

71.406% 

76.251% 

80.135% 

83.318% 

85.974% 

88.224% 

90.154% 

91.828% 

93.294% 

94.589% 

95.740% 

96.771% 

97.699% 

98.539% 

99.302% 

8 

34.457% 

52.603% 

63.804% 

71.406% 

76.903% 

81.064% 

84.322% 

86.944% 

89.098% 

90.900% 

92.429% 

93.743% 

94.885% 

95.886% 

96.771% 

97.558% 

98.264% 

98.900% 

99.476% 

9 

41.210% 

59.674% 

70.152% 

76.903% 

81.616% 

85.092% 

87.762% 

89.877% 

91.594% 

93.016% 

94.212% 

95.233% 

96.114% 

96.882% 

97.558% 

98.157% 

98.692% 

99.173% 

99.606% 

10 

48.310% 

66.365% 

75.809% 

81.616% 

85.548% 

88.386% 

90.532% 

92.211% 

93.560% 

94.669% 

95.595% 

96.382% 

97.057% 

97.643% 

98.157% 

98.612% 

99.016% 

99.378% 

99.704% 

11 

55.479% 

72.458% 

80.689% 

85.548% 

88.754% 

91.029% 

92.727% 

94.042% 

95.091% 

95.948% 

96.660% 

97.261% 

97.776% 

98.222% 

98.612% 

98.955% 

99.260% 

99.533% 

99.778% 

12 

62.428% 

77.816% 

84.782% 

88.754% 

91.322% 

93.118% 

94.444% 

95.464% 

96.273% 

96.930% 

97.474% 

97.932% 

98.323% 

98.661% 

98.955% 

99.214% 

99.444% 

99.649% 

99.834% 

13 

68.900% 

82.385% 

88.135% 

91.322% 

93.347% 

94.748% 

95.774% 

96.559% 

97.178% 

97.679% 

98.093% 

98.441% 

98.737% 

98.992% 

99.214% 

99.410% 

99.583% 

99.737% 

99.875% 

14 

74.708% 

86.180% 

90.829% 

93.347% 

94.926% 

96.008% 

96.797% 

97.397% 

97.869% 

98.249% 

98.563% 

98.826% 

99.050% 

99.242% 

99.410% 

99.557% 

99.687% 

99.802% 

99.906% 

15 

79.751% 

89.264% 

92.960% 

94.926% 

96.145% 

96.976% 

97.578% 

98.035% 

98.393% 

98.681% 

98.918% 

99.117% 

99.286% 

99.431% 

99.557% 

99.667% 

99.765% 

99.852% 

99.930% 

16 

84.003% 

91.726% 

94.626% 

96.145% 

97.081% 

97.715% 

98.173% 

98.519% 

98.790% 

99.008% 

99.187% 

99.336% 

99.463% 

99.572% 

99.667% 

99.750% 

99.823% 

99.889% 

99.947% 

17 

87.503% 

93.663% 

95.914% 

97.081% 

97.795% 

98.276% 

98.623% 

98.885% 

99.090% 

99.254% 

99.389% 

99.501% 

99.597% 

99.679% 

99.750% 

99.812% 

99.868% 

99.917% 

99.960% 

18 

90.325% 

95.171% 

96.904% 

97.795% 

98.337% 

98.702% 

98.964% 

99.161% 

99.316% 

99.439% 

99.541% 

99.626% 

99.697% 

99.759% 

99.812% 

99.859% 

99.901% 

99.937% 

99.970% 

19 

92.564% 

96.334% 

97.660% 

98.337% 

98.747% 

99.023% 

99.221% 

99.370% 

99.486% 

99.579% 

99.655% 

99.719% 

99.773% 

99.819% 

99.859% 

99.894% 

99.925% 

99.953% 

99.978% 

20 

94.317% 

97.225% 

98.235% 

98.747% 

99.058% 

99.265% 

99.415% 

99.527% 

99.614% 

99.684% 

99.741% 

99.789% 

99.830% 

99.864% 

99.894% 

99.921% 

99.944% 

99.965% 

99.983% 

21 

95.676% 

97.904% 

98.670% 

99.058% 

99.292% 

99.448% 

99.560% 

99.645% 

99.710% 

99.763% 

99.806% 

99.842% 

99.872% 

99.898% 

99.921% 

99.941% 

99.958% 

99.974% 

99.987% 

22 

96.722% 

98.420% 

98.999% 

99.292% 

99.468% 

99.586% 

99.670% 

99.733% 

99.782% 

99.822% 

99.854% 

99.881% 

99.904% 

99.924% 

99.941% 

99.955% 

99.969% 

99.980% 

99.991% 

23 

97.521% 

98.810% 

99.248% 

99.468% 

99.600% 

99.689% 

99.752% 

99.800% 

99.837% 

99.866% 

99.891% 

99.911% 

99.928% 

99.943% 

99.955% 

99.967% 

99.976% 

99.985% 

99.993% 

24 

98.129% 

99.105% 

99.435% 

99.600% 

99.700% 

99.766% 

99.814% 

99.850% 

99.878% 

99.900% 

99.918% 

99.933% 

99.946% 

99.957% 

99.967% 

99.975% 

99.982% 

99.989% 

99.995% 

25 

98.590% 

99.327% 

99.575% 

99.700% 

99.775% 

99.825% 

99.860% 

99.887% 

99.908% 

99.925% 

99.938% 

99.950% 

99.959% 

99.968% 

99.975% 

99.981% 

99.987% 

99.992% 

99.996% 

26 

98.939% 

99.495% 

99.681% 

99.775% 

99.831% 

99.868% 

99.895% 

99.915% 

99.931% 

99.944% 

99.954% 

99.962% 

99.970% 

99.976% 

99.981% 

99.986% 

99.990% 

99.994% 

99.997% 

27 

99.202% 

99.620% 

99.761% 

99.831% 

99.873% 

99.901% 

99.921% 

99.937% 

99.948% 

99.958% 

99.965% 

99.972% 

99.977% 

99.982% 

99.986% 

99.989% 

99.993% 

99.995% 

99.998% 

28 

99.400% 

99.715% 

99.820% 

99.873% 

99.905% 

99.926% 

99.941% 

99.952% 

99.961% 

99.968% 

99.974% 

99.979% 

99.983% 

99.986% 

99.989% 

99.992% 

99.994% 

99.996% 

99.998% 

29 

99.550% 

99.786% 

99.865% 

99.905% 

99.929% 

99.944% 

99.956% 

99.964% 

99.971% 

99.976% 

99.981% 

99.984% 

99.987% 

99.990% 

99.992% 

99.994% 

99.996% 

99.997% 

99.9feS% 

30 

99.662% 

99.840% 

99.899% 

99.929% 

99.946% 

99.958% 

99.967% 

99.973% 

99.978% 

99.982% 

99.985% 

99.988% 

99.990% 

99.992% 

99.994% 

99.996% 

99.997% 

99.998% 

99.999% 


10/2010 ( 15 ) 


Binomial-Based Test Requirements Are Generally Much Too Large 
Compared to Reliability-Growth-Based Test Requirements 


Number of 
Tests 

Binomial 

(50%) 

Initial Assurance Level: Test Effectiveness=25% 

5% 

10% 

25% 

50% 

75% 

90% 

95% 

1 

25 . 000 % 

6 . 557 % 

12 . 903 % 

30 . 769 % 

57 . 143 % 

80 . 000 % 

92 . 308 % 

96 . 203 % 

2 

50 . 000 % 

8 . 556 % 

16 . 495 % 

37 . 209 % 

64 . 000 % 

84 . 211 % 

94 . 118 % 

97 . 125 % 

3 

62 . 996 % 

11 . 092 % 

20 . 847 % 

44 . 138 % 

70 . 330 % 

87 . 671 % 

95 . 522 % 

97 . 828 % 

4 

70 . 711 % 

14 . 262 % 

25 . 990 % 

51 . 303 % 

75 . 964 % 

90 . 459 % 

96 . 604 % 

98 . 362 % 

5 

75 . 786 % 

18 . 153 % 

31 . 890 % 

58 . 414 % 

80 . 821 % 

92 . 670 % 

97 . 431 % 

98 . 766 % 

6 

79 . 370 % 

22 . 823 % 

38 . 435 % 

65 . 192 % 

84 . 891 % 

94 . 400 % 

98 . 061 % 

99 . 072 % 

7 

82 . 034 % 

28 . 279 % 

45 . 427 % 

71 . 406 % 

88 . 224 % 

95 . 740 % 

98 . 539 % 

99 . 302 % 

8 

84 . 090 % 

34 . 457 % 

52 . 603 % 

76 . 903 % 

90 . 900 % 

96 . 771 % 

98 . 900 % 

99 . 476 % 

9 

85 . 724 % 

41 . 210 % 

59 . 674 % 

81 . 616 % 

93 . 016 % 

97 . 558 % 

99 . 173 % 

99 . 606 % 

10 

87 . 055 % 

48 . 310 % 

66 . 365 % 

85 . 548 % 

94 . 669 % 

98 . 157 % 

99 . 378 % 

99 . 704 % 

11 

88 . 159 % 

55 . 479 % 

72 . 458 % 

88 . 754 % 

95 . 948 % 

98 . 612 % 

99 . 533 % 

99 . 778 % 

12 

89 . 090 % 

62 . 428 % 

77 . 816 % 

91 . 322 % 

96 . 930 % 

98 . 955 % 

99 . 649 % 

99 . 834 % 

13 

89 . 885 % 

68 . 900 % 

82 . 385 % 

93 . 347 % 

97 . 679 % 

99 . 214 % 

99 . 737 % 

99 . 875 % 

14 

90 . 572 % 

74 . 708 % 

86 . 180 % 

94 . 926 % 

98 . 249 % 

99 . 410 % 

99 . 802 % 

99 . 906 % 

15 

91 . 172 % 

79 . 751 % 

89 . 264 % 

96 . 145 % 

98 . 681 % 

99 . 557 % 

99 . 852 % 

99 . 930 % 

16 

91 . 700 % 

84 . 003 % 

91 . 726 % 

97 . 081 % 

99 . 008 % 

99 . 667 % 

99 . 889 % 

99 . 947 % 

17 

92 . 169 % 

87 . 503 % 

93 . 663 % 

97 . 795 % 

99 . 254 % 

99 . 750 % 

99 . 917 % 

99 . 960 % 

18 

92 . 587 % 

90 . 325 % 

95 . 171 % 

98 . 337 % 

99 . 439 % 

99 . 812 % 

99 . 937 % 

99 . 970 % 

19 

92 . 964 % 

92 . 564 % 

96 . 334 % 

98 . 747 % 

99 . 579 % 

99 . 859 % 

99 . 953 % 

99 . 978 % 

20 

93 . 303 % 

94 . 317 % 

97 . 225 % 

99 . 058 % 

99 . 684 % 

99 . 894 % 

99 . 965 % 

99 . 983 % 

21 

93 . 612 % 

95 . 676 % 

97 . 904 % 

99 . 292 % 

99 . 763 % 

99 . 921 % 

99 . 974 % 

99 . 987 % 

22 

93 . 893 % 

96 . 722 % 

98 . 420 % 

99 . 468 % 

99 . 822 % 

99 . 941 % 

99 . 980 % 

99 . 991 % 

23 

94 . 151 % 

97 . 521 % 

98 . 810 % 

99 . 600 % 

99 . 866 % 

99 . 955 % 

99 . 985 % 

99 . 993 % 

24 

94 . 387 % 

98 . 129 % 

99 . 105 % 

99 . 700 % 

99 . 900 % 

99 . 967 % 

99 . 989 % 

99 . 995 % 

25 

94 . 606 % 

98 . 590 % 

99 . 327 % 

99 . 775 % 

99 . 925 % 

99 . 975 % 

99 . 992 % 

99 . 996 % 

26 

94 . 808 % 

98 . 939 % 

99 . 495 % 

99 . 831 % 

99 . 944 % 

99 . 981 % 

99 . 994 % 

99 . 997 % 

27 

94 . 995 % 

99 . 202 % 

99 . 620 % 

99 . 873 % 

99 . 958 % 

99 . 986 % 

99 . 995 % 

99 . 998 % 

28 

95 . 170 % 

99 . 400 % 

99 . 715 % 

99 . 905 % 

99 . 968 % 

99 . 989 % 

99 . 996 % 

99 . 998 % 

29 

95 . 332 % 

99 . 550 % 

99 . 786 % 

99 . 929 % 

99 . 976 % 

99 . 992 % 

99 . 997 % 

99 . 999 % 

30 

95 . 484 % 

99 . 662 % 

99 . 840 % 

99 . 946 % 

99 . 982 % 

99 . 994 % 

99 . 998 % 

99 . 999 % 
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Reliability-Growth-Based Failure-Free Tests to Demonstrate a Given 
Design Reliability Versus Fault Detection Effectiveness 


Probability That a System is Free of Failures After a Given Number of Failure-Free Tests Have Been Conducted With Each Test Having a Given 
Detection Effectiveness For a Given Initial System Assurance Level 


Initial System Assurance 
Level 


30 % 


Detection Effectiveness Value 


1 

Alternative Detection 
Effectiveness Values 
Number of Failure-Free 
Tests Conducted 

1 

5% 

10% 

15% 

20% 

25% 

30% 

35% 

40% 

45% 

50% 

55% 

60% 

65% 

70% 

75% 

80% 

85% 

90% 

95% 

Probability the System is Failure-Free After Conducting a Given Number of Failure-Free Tests 

31 . 0881 % 

32 . 2581 % 

33 . 5196 % 

34 . 8837 % 

36 . 3636 % 

37 . 9747 % 

39 . 7351 % 

41 . 6667 % 

43 . 7956 % 

46 . 1538 % 

48 . 7805 % 

51 . 7241 % 

55 . 0459 % 

58 . 8235 % 

63 . 1579 % 

68 . 1818 % 

74 . 0741 % 

81 . 0811 % 

89 . 5522 % 

2 

32 . 1975 % 

34 . 6021 % 

37 . 2324 % 

40 . 1070 % 

43 . 2432 % 

46 . 6563 % 

50 . 3567 % 

54 . 3478 % 

58 . 6224 % 

63 . 1579 % 

67 . 9117 % 

72 . 8155 % 

77 . 7706 % 

82 . 6446 % 

87 . 2727 % 

91 . 4634 % 

95 . 0119 % 

97 . 7199 % 

99 . 4200 % 

3 

33 . 3273 % 

37 . 0233 % 

41 . 1022 % 

45 . 5650 % 

50 . 3937 % 

55 . 5453 % 

60 . 9462 % 

66 . 4894 % 

72 . 0353 % 

77 . 4194 % 

82 . 4657 % 

87 . 0070 % 

90 . 9056 % 

94 . 0734 % 

96 . 4824 % 

98 . 1675 % 

99 . 2187 % 

99 . 7672 % 

99 . 9708 % 

4 

34 . 4766 % 

39 . 5116 % 

45 . 0854 % 

51 . 1317 % 

57 . 5281 % 

64 . 0930 % 

70 . 5958 % 

76 . 7813 % 

82 . 4053 % 

87 . 2727 % 

91 . 2674 % 

94 . 3634 % 

96 . 6170 % 

98 . 1451 % 

99 . 0968 % 

99 . 6281 % 

99 . 8820 % 

99 . 9767 % 

99 . 9985 % 

5 

35 . 6444 % 

42 . 0555 % 

49 . 1325 % 

56 . 6705 % 

64 . 3620 % 

71 . 8307 % 

78 . 6946 % 

84 . 6425 % 

89 . 4908 % 

93 . 2039 % 

95 . 8721 % 

97 . 6664 % 

98 . 7893 % 

99 . 4362 % 

99 . 7727 % 

99 . 9254 % 

99 . 9823 % 

99 . 9977 % 

99 . 9999 % 

6 

36 . 8295 % 

44 . 6423 % 

53 . 1910 % 

62 . 0475 % 

70 . 6572 % 

78 . 4613 % 

85 . 0356 % 

90 . 1824 % 

93 . 9330 % 

96 . 4824 % 

98 . 0993 % 

99 . 0533 % 

99 . 5729 % 

99 . 8302 % 

99 . 9431 % 

99 . 9851 % 

99 . 9973 % 

99 . 9998 % 

100 . 0000 % 

7 

38 . 0307 % 

47 . 2584 % 

57 . 2078 % 

67 . 1440 % 

76 . 2508 % 

83 . 8814 % 

89 . 7356 % 

93 . 8686 % 

96 . 5695 % 

98 . 2097 % 

99 . 1356 % 

99 . 6192 % 

99 . 8501 % 

99 . 9490 % 

99 . 9858 % 

99 . 9970 % 

99 . 9996 % 

100 . 0000 % 

100 . 0000 % 

8 

39 . 2468 % 

49 . 8897 % 

61 . 1317 % 

71 . 8665 % 

81 . 0638 % 

88 . 1436 % 

93 . 0795 % 

96 . 2287 % 

98 . 0836 % 

99 . 0968 % 

99 . 6092 % 

99 . 8473 % 

99 . 9475 % 

99 . 9847 % 

99 . 9964 % 

99 . 9994 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

9 

40 . 4763 % 

52 . 5215 % 

64 . 9165 % 

76 . 1513 % 

85 . 0921 % 

91 . 3944 % 

95 . 3900 % 

97 . 7026 % 

98 . 9368 % 

99 . 5463 % 

99 . 8238 % 

99 . 9389 % 

99 . 9816 % 

99 . 9954 % 

99 . 9991 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

10 

41 . 7179 % 

55 . 1395 % 

68 . 5225 % 

79 . 9655 % 

88 . 3862 % 

93 . 8165 % 

96 . 9544 % 

98 . 6088 % 

99 . 4125 % 

99 . 7727 % 

99 . 9206 % 

99 . 9755 % 

99 . 9936 % 

99 . 9986 % 

99 . 9998 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

11 

42 . 9701 % 

57 . 7292 % 

71 . 9183 % 

83 . 3034 % 

91 . 0292 % 

95 . 5897 % 

97 . 9990 % 

99 . 1606 % 

99 . 6760 % 

99 . 8862 % 

99 . 9643 % 

99 . 9902 % 

99 . 9977 % 

99 . 9996 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

12 

44 . 2313 % 

60 . 2772 % 

75 . 0809 % 

86 . 1812 % 

93 . 1175 % 

96 . 8714 % 

98 . 6902 % 

99 . 4947 % 

99 . 8215 % 

99 . 9431 % 

99 . 9839 % 

99 . 9961 % 

99 . 9992 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

13 

45 . 5001 % 

62 . 7706 % 

77 . 9963 % 

88 . 6308 % 

94 . 7478 % 

97 . 7892 % 

99 . 1447 % 

99 . 6962 % 

99 . 9018 % 

99 . 9715 % 

99 . 9928 % 

99 . 9984 % 

99 . 9997 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

14 

46 . 7747 % 

65 . 1979 % 

80 . 6584 % 

90 . 6930 % 

96 . 0084 % 

98 . 4421 % 

99 . 4424 % 

99 . 8175 % 

99 . 9459 % 

99 . 9858 % 

99 . 9967 % 

99 . 9994 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

15 

48 . 0535 % 

67 . 5487 % 

83 . 0684 % 

92 . 4132 % 

96 . 9761 % 

98 . 9044 % 

99 . 6368 % 

99 . 8904 % 

99 . 9703 % 

99 . 9929 % 

99 . 9985 % 

99 . 9997 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

16 

49 . 3349 % 

69 . 8143 % 

85 . 2331 % 

93 . 8370 % 

97 . 7148 % 

99 . 2305 % 

99 . 7636 % 

99 . 9342 % 

99 . 9836 % 

99 . 9964 % 

99 . 9993 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

17 

50 . 6172 % 

71 . 9873 % 

87 . 1638 % 

95 . 0081 % 

98 . 2763 % 

99 . 4601 % 

99 . 8462 % 

99 . 9605 % 

99 . 9910 % 

99 . 9982 % 

99 . 9997 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

18 

51 . 8986 % 

74 . 0620 % 

88 . 8751 % 

95 . 9662 % 

98 . 7016 % 

99 . 6215 % 

99 . 9000 % 

99 . 9763 % 

99 . 9951 % 

99 . 9991 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

19 

53 . 1776 % 

76 . 0341 % 

90 . 3833 % 

96 . 7467 % 

99 . 0230 % 

99 . 7347 % 

99 . 9350 % 

99 . 9858 % 

99 . 9973 % 

99 . 9996 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

20 

54 . 4524 % 

77 . 9011 % 

91 . 7062 % 

97 . 3803 % 

99 . 2655 % 

99 . 8142 % 

99 . 9577 % 

99 . 9915 % 

99 . 9985 % 

99 . 9998 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

21 

55 . 7214 % 

79 . 6615 % 

92 . 8615 % 

97 . 8932 % 

99 . 4481 % 

99 . 8698 % 

99 . 9725 % 

99 . 9949 % 

99 . 9992 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

22 

56 . 9829 % 

81 . 3154 % 

93 . 8666 % 

98 . 3074 % 

99 . 5855 % 

99 . 9089 % 

99 . 9821 % 

99 . 9969 % 

99 . 9995 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

23 

58 . 2355 % 

82 . 8636 % 

94 . 7382 % 

98 . 6414 % 

99 . 6888 % 

99 . 9362 % 

99 . 9884 % 

99 . 9982 % 

99 . 9998 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

24 

59 . 4775 % 

84 . 3084 % 

95 . 4919 % 

98 . 9101 % 

99 . 7664 % 

99 . 9553 % 

99 . 9925 % 

99 . 9989 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

25 

60 . 7075 % 

85 . 6524 % 

96 . 1420 % 

99 . 1262 % 

99 . 8247 % 

99 . 9687 % 

99 . 9951 % 

99 . 9993 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

26 

61 . 9241 % 

86 . 8992 % 

96 . 7016 % 

99 . 2997 % 

99 . 8685 % 

99 . 9781 % 

99 . 9968 % 

99 . 9996 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

27 

63 . 1259 % 

88 . 0528 % 

97 . 1824 % 

99 . 4390 % 

99 . 9013 % 

99 . 9847 % 

99 . 9979 % 

99 . 9998 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

28 

64 . 3116 % 

89 . 1175 % 

97 . 5949 % 

99 . 5507 % 

99 . 9260 % 

99 . 9893 % 

99 . 9987 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

29 

65 . 4800 % 

90 . 0980 % 

97 . 9483 % 

99 . 6402 % 

99 . 9445 % 

99 . 9925 % 

99 . 9991 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

30 

66 . 6301 % 

90 . 9990 % 

98 . 2506 % 

99 . 7120 % 

99 . 9583 % 

99 . 9947 % 

99 . 9994 % 

99 . 9999 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 

100 . 0000 % 
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Reliability Distribution After a Given Number of Failure- 
Free Tests for the Large Uncertainty Case 





Reliability Distribution Characteristics After a Given Number of Tests 



ITest 

2 Tests 

3 Tests 

4 Tests 

5 Tests 

10 Tests 

15 Tests 

20 Tests 

1 % 

14.075509% 

16.811989% 

19.535827% 

22.401186% 

25.350151% 

41.706695% 

58.565376% 

73.192121% 

2.50% 

16.223551% 

20.168430% 

23.989777% 

27.843530% 

31.741291% 

51.959003% 

69.602750% 

82.386888% 

5% 

18.972912% 

24.312700% 

29.469920% 

34.532614% 

39.569888% 

62.489285% 

79.163676% 

89.124958% 

50% 

59.173562% 

67.731535% 

75.264301% 

81.472889% 

86.370113% 

97.417663% 

99.544091% 

99.922318% 

95% 

89.990637% 

93.176128% 

95.683576% 

97.403847% 

98.485325% 

99.917874% 

99.996223% 

99.999837% 

97.50% 

91.522460% 

94.582432% 

96.760481% 

98.130984% 

98.951206% 

99.950266% 

99.997926% 

99.999918% 

99% 

92.754334% 

95.688253% 

97.549978% 

98.653284% 

99.271237% 

99.969511% 

99.998842% 

99.999957% 

Mean 

57.238016% 

64.074271% 

70.213039% 

75.477049% 

79.833896% 

91.788449% 

96.173147% 

98.105062% 
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Mission Success Starts With Safety 

Handling Failures That Occur 


* A failure occurrence can be handled by discounting 
the failure in the reliability estimation 

* The discount factor is one minus the corrective action 
effectiveness (the ineffectiveness) 

* A failure occurrence can alternatively be handled by 
restarting the reliability at the value before the failure 
multiplied by the corrective action effectiveness 

* Both alternatives give similar results 

* With the restart alternative the reliability-growth test 
tables can be entered with the restarted reliability 
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Using a Dynamic Reliability Model to Monitor the 
Dynamic Reliability Growth of a Spacecraft 


• The following slide shows an example evaluation of the 
dynamic tracking of a hypothetical flight history 

• The evaluations incorporate fault removal and include the 
random operational contribution 

• The values are Kalman-Filter-predicted next-flight reliability 
based on the past history up to the flight 

• The spikes on the curves are the predicted reliability after 
failure fixes have been made and have been included 

• Such monitoring evaluations are important for tracking real- 
time reliability growth to update analyses and actions 
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Reliability-Growth-Based Model for the Probability of Zero 
Failure-Causing Faults Existing After a Given Number of 
Successful Tests 


P(0) = initial probability of nofailure causing faults in the design (initial assurance level) 

7V = number of failure free tests or flights conducted 

p = fault detection effectiveness ( conditional probability of detecting a failure -causing fault) 

P(0/N)= probability of no failure causingfaults in the design after N failure free tests 
Using Bayes theorem 


P(0/N ) 


m 

P(0) + (l-P(0))(l-pf 


P( 0 / N) Is also termed the design reliability and is calculated in the previous tables 



Mission Success Starts With Safety 

Extensions of the Formula for No Failure- 
Causing Faults Existing (1) 


The formula previously was for a single fault existing. For multiple faults 
existing the formula becomes 


where 


P(0/N) 


m 

P(0) + ^P(k)P(0/k,N) 

k 


p(k) = the probability of k faults existing 


P(0I k,N) - the probability of missing all k faults in N tests 


The sum in the denominator is over k faults existing with k greater than 
or equal to 1. 
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Mission Success Starts With Safety 

Extensions of the Formula for No Failure- 
Causing Faults Existing (2) 


For a standard test which is less likely to miss all faults if more than one exists we 
have 


P(0lk,N)<P(0l\,N) 


Hence, 


P(0/N)> 


m 

p(0) + (£p(k))(P(on,N)) 

k 


P(0) 

P(0) + (l-P(0))(l-p) N 


which is the formula in the main body. 

Consequently if multiple faults exist the formula for one fault gives a lower 
bound on the reliability, i.e. , on P{ 0/ N) and the reliability 
can be somewhat higher 
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Mission Success Starts With Safety 

Extensions of the Formula for No Failure- 
Causing Faults Existing (3) 


If P(k ) follows a Poisson then it is straightforward to show that 

P(0/A0 = exp(-A(l-/?) A ') 

where 


\ - the expected number of faults existing 


If the tests are repetitive or are correlated with a portion of the conditions 
repeated then N is replaced by the effective number of non-overlapping tests 
conducted. In the case of overlapping tests where N is replaced by log N then the 
model is similar to the Duane model where now the parameters are expressed in 
terms of the expected number of faults existing and the fault detection coverage. 
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